The embodiments described herein pertain in general to conducting controlled, randomized experiments and more particularly to implementing a reset policy during a variation test of content.
Content variation testing is an experiment that tests changes to content against another version of the content. Content variation testing is applied to many different internet services such as electronic-commerce sites to social networks to market places. One example of content variation testing is web page variation testing where an experiment is conducted that tests changes to a web page against another design of the web page. During variation testing of a web page, changes to a web page that produce positive results are determined. The positive results associated with a particular change to a web page validates that the change to the web page should be used.
To obtain valid results in a conventional variation test such as a fixed-horizon variation test, a strict set of guidelines are followed when performing the variation test. The guidelines include setting a minimum detectable effect and sample size in advance to conducting the test, refraining from viewing the results of the variation test prior to the completion of the variation test, and refraining from testing too many goals and variations in the variation test.
However, an experimenter can now continuously monitor the results of a variation test due to the advancement of experimental technology. This poses a problem as conventional variation testing requires that the experimenter refrain from viewing the results until the completion of the variation test. Thus, the experimenter may view invalid results if the experimenter views the results of the variation test before the completion of the variation test and may make decisions based on those invalid results.